Predicting optimal values for parameters used in an operation of an image signal processor using machine learning

ABSTRACT

A method of predicting optimal values for a plurality of parameters used in an operation of an image signal processor includes: inputting initial values for the plurality of parameters to a machine learning model having an input layer, corresponding to the plurality of parameters, and an output layer corresponding to a plurality of evaluation items extracted from a result image generated by the image signal processor; obtaining evaluation scores for the plurality of evaluation items using an output of the machine learning model; adjusting weights, applied to the plurality of parameters, based on the evaluation scores; and determining the optimal values using the adjusted weights.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This U.S. non-provisional patent application claims the benefit ofpriority under 35 U.S.C. § 119 to Korean Patent Application No.10-2019-0059573 filed on May 21, 2019 in the Korean IntellectualProperty Office, the disclosure of which is incorporated by reference inits entirety herein.

BACKGROUND 1. Technical Field

The present disclosure relates to a method of training a machinelearning to predict optimal values for parameters used in an operationof an image signal processor and an electronic device configured toperform the method.

2. Discussion of Related Art

An image sensor is a semiconductor-based sensor configured to receivelight and generate an electrical signal. Raw data, output by the imagesensor, may be processed by an image signal processor (ISP). The imagesignal processor may generate an image using the raw data output by theimage sensor. The image signal processor may generate an image from theraw data based on various parameters. However, quality andcharacteristics of the generated image may vary depending on values ofthe parameters applied to the image signal processor.

SUMMARY

At least one exemplary embodiment of the inventive concept provides amethod predicting performance of an image signal processor or quality ofimages generated by the image signal using machine learning. Theresulting predictions may be used to tune the image signal processor toimprove quality of images generated by the image signal processor.

According to an exemplary embodiment of the inventive concept, a methodof training a machine learning model to predict optimal values for aplurality of parameters used in an operation of an image signalprocessor includes: capturing an image of a sample subject to obtainsample data; generating a plurality of sets of sample values for theplurality of parameters; emulating the image signal processor (ISP)processing the sample data according to each of the sets to generate aplurality of sample images; evaluating each of the plurality of sampleimages for a plurality of evaluation items to generate respective samplescores; and training the machine learning model to predict the optimalvalues using the sample values and the sample scores.

According to an exemplary embodiment of the inventive concept, a methodof predicting optimal values for a plurality of parameters used in anoperation of an image signal processor includes: inputting initialvalues for the plurality of parameters to a machine learning modelincluding an input layer having a plurality of input nodes,corresponding to the plurality of parameters, and an output layer havinga plurality of output nodes, corresponding to a plurality of evaluationitems extracted from a result image generated by the image signalprocessor; obtaining evaluation scores for the plurality of evaluationitems using an output of the machine learning model; adjusting weights,applied to the plurality of parameters, based on the evaluation scores;and determining the optimal values using the adjusted weights.

According to an exemplary embodiment of the inventive concept, anelectronic device includes an image signal processor and a parameteroptimization module. The image signal processor is configured to processraw data, output by an image sensor, depending on a plurality ofparameters to generate a result image. The parameter optimization moduleincludes a machine learning model, receiving sample values for theplurality of parameters and outputting a plurality of sample scoresindicating quality of sample images, the sample images being generatedby the image signal processor processing the raw data based on thesample values, the parameter optimization module being configured todetermine weights, respectively applied to the plurality of parameters,using the machine learning model. The image signal processor applies theweights to the plurality of parameters to generate a plurality ofweighted parameters and generates the result image by processing the rawdata using the weighted parameters.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments of the present disclosure will be more clearly understoodfrom the following detailed description, taken in conjunction with theaccompanying drawings, in which:

FIG. 1 is a block diagram of an image sensor according to an exampleembodiment;

FIGS. 2 and 3 are schematic diagrams of image sensors according to anexample embodiment, respectively;

FIG. 4 illustrates a pixel array of an image sensor according to anexample embodiment;

FIG. 5 is a block diagram of an electronic device according to anexemplary embodiment of the inventive concept;

FIG. 6 is a flowchart illustrating a method of generating data that maybe used to train a machine learning model for an image signal processoraccording to an exemplary embodiment of the inventive concept;

FIG. 7 illustrates a system that may use the machine learning modelaccording to an exemplary embodiment of the inventive concept;

FIGS. 8 to 10 illustrate a method of training the machine learning modelaccording to an exemplary embodiment of the inventive concept;

FIG. 11 illustrates a system for training the machine learning modelaccording to an exemplary embodiment of the inventive concept;

FIG. 12 illustrates the machine learning model according to an exemplaryembodiment of the inventive concept;

FIG. 13 is a flowchart illustrating a method of operating the machinelearning model according to an exemplary embodiment of the inventiveconcept;

FIG. 14 illustrates a system providing a method of operating the machinelearning model according to an exemplary embodiment of the inventiveconcept;

FIGS. 15 to 17 illustrate a method of operating the machine learningmodel according to an exemplary embodiment of the inventive concept;

FIG. 18 is a block diagram of an electronic device according to anexemplary embodiment of the inventive concept;

FIGS. 19A and 19B illustrate an electronic device according to anexemplary embodiment of the inventive concept; and

FIGS. 20 and 21 illustrate an operation of an electronic deviceaccording to an exemplary embodiment of the inventive concept.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

Hereinafter, exemplary embodiments of the inventive concept will bedescribed with reference to the accompanying drawings.

FIG. 1 is a block diagram of an image sensor according to an exemplaryembodiment of the inventive concept.

Referring to FIG. 1, an image sensor 10 according to an exemplaryembodiment includes a pixel array 11, a row driver 12 (e.g., a rowdriving circuit), a readout circuit 13, and a column driver 14 (e.g., acolumn driving circuit), control logic 15 (e.g., a logic or controlcircuit). The row driver 12, the readout circuit 13, the column driver14, and the control logic 16 may be circuits configured to generateimage data for controlling the pixel array 11, and may be incorporatedinto a controller.

The image sensor 10 may convert light, transferred from an object 30,into an electrical signal to generate raw data for generating image. Theraw data may be output to a processor 20. The processor 20 may includean image signal processor (ISP) configured to generate an image usingthe raw data. According to an exemplary embodiment of the inventiveconcept, the image signal processor is mounted in the image sensor 10.

The pixel array 11, incorporated in the image sensor 10, may include aplurality of pixels PX. Each of the plurality of pixels PX may includean optoelectronic component configured to receive light and generatecharges based on the received light, for example, a photodiode (PD). Inan exemplary embodiment, each of the plurality of pixels PX includes twoor more optoelectronic components. Two or more optoelectronic componentsmay be included in each of the plurality of pixels PX such that each ofthe pixels PX generates a pixel signal corresponding to light of variouswavelength bands or provides an autofocusing function.

Each of the plurality of pixels PX may include a pixel circuitconfigured to generate a pixel signal from charges generated by one ormore photodiodes. In an exemplary embodiment, the pixel circuit includesa transmission transistor, a drive transistor, a select transistor, anda reset transistor. As an example, the pixel circuit may output a resetvoltage and a pixel voltage using charges generated by the photodiodes.The pixel voltage may be a voltage reflecting charges generated byphotodiodes included in each of the plurality of pixels PX. In anexemplary embodiment, two or more adjacent pixels PX may constitute asingle pixel group, and two or more pixels, belonging to a pixel group,may share at least some of a transmission transistor, a drivetransistor, a select transistor, and a reset transistor with each other.

The row driver 12 may drive the pixel array 11 in units of rows. Forexample, the row driver 12 may generate a transmission control signalcontrolling a transmission transistor of a pixel circuit, a resetcontrol signal controlling a reset transistor of the pixel circuit, anda select control signal controlling a select transistor of the pixelcircuit.

The readout circuit 13 may include at least one of a correlated doublesampler (CDS) and an analog-to-digital converter (ADC). The correlateddouble sampler may be connected to pixels, included in a row lineselected by a row select signal provided by the row driver 12, throughcolumn lines and may perform correlated double sampling to detect areset voltage and a pixel voltage. The analog-to-digital converter mayoutput a digital signal after converting the reset voltage and the pixelvoltage, detected by the correlated double sampler, into the digitalsignal.

The column driver 14 may include a latch circuit, a buffer, an amplifiercircuit, and may temporarily store or amplify the digital signal,received from the readout circuit 13, to generate image data. Operatingtimings of the row driver 12, the readout circuit 13, and the columndriver 14 may be determined by the control logic 15. As an example, thecontrol logic 15 may be operated by a control instruction transmitted bythe processor 20. The processor 20 may signal-process the raw data,output by the column driver 14 and the control logic 15, to generate animage and may output the image to a display device, or store the imagein a storage device such as a memory.

FIGS. 2 and 3 are schematic diagrams of image sensors according toexemplary embodiments, respectively.

Referring to FIG. 2, an image sensor 40 according to an exemplaryembodiment includes a first layer 41, a second layer 42 provided belowthe first layer 41, and a third layer 43 provided below the second layer42. The first layer 41, the second layer 42, and the third layer 43 maybe stacked in a direction perpendicular to each other. In an exemplaryembodiment, the first layer 41 and the second layer 42 are stacked in awafer level, and the third layer 43 is attached to a portion below thesecond layer 42. The first to third layers 41, 42, and 43 may beprovided as a single semiconductor package.

The first layer 41 includes a sensing area SA, in which a plurality ofpixels are provided, and a first pad area PA1 provided around thesensing area SA. A plurality of upper pads PAD are included in the firstpad region PAL The plurality of upper pads PAD may be connected to padsand a logic circuit LC of the second layer 42 through a via or a wire.The pads of the second layer 42 may be provided in a second pad area PA2of the second layer 42.

Each of the plurality of pixels PX may include a photodiode configuredto receive light and generate charges and a pixel circuit configured toprocess the charges generated by the photodiode. The pixel circuit mayinclude a plurality of transistors configured to output a voltagecorresponding to the charges generated by a photodiode.

The second layer 42 may include a plurality of components configured toimplement the control logic LC. The plurality of components implementingthe logic circuit LC may include circuits configured to drive a pixelcircuit provided on the first layer 41, such as a row driver, a readoutcircuit, a column driver, and control logic. The plurality of componentsimplementing the logic circuit LC may be connected to a pixel circuitthrough the first and second pad areas PA1 and PA2. The logic circuit LCmay obtain the reset voltage and the pixel voltage from the plurality ofpixels PX to generate a pixel signal.

In an exemplary embodiment, at least one of the plurality of pixels PXincludes a plurality of photodiodes disposed on the same level. Pixelsignals, generated from charges of each of the plurality of photodiodes,may have a phase difference from each other. The logic circuit LC mayprovide an autofocusing function based on a phase difference of pixelsignals generated from a plurality of photodiodes included in a singlepixel PX.

The third layer 43, provided below the second layer 42, may include amemory chip MC, a dummy chip DC and an encapsulation layer ENencapsulating the memory chip MC and the dummy chip DC. The memory chipMC may be a dynamic random access memory (DRAM) or a static randomaccess memory (SRAM). In an embodiment, the dummy chip DC does not storedata. The dummy chip DC may be omitted. The memory chip MC may beelectrically connected to at least some of the components, included inthe logic circuit LC of the second layer 42, by a bump, a via, or awire, and may store data required to provide an autofocusing function.In an exemplary embodiment, the bump is a microbump.

Referring to FIG. 3, an image sensor 50 according to an exemplaryembodiment includes a first layer 51 and a second layer 52. The firstlayer 51 includes a sensing area SA in which a plurality of pixels PXare provided, a logic circuit LC in which components for driving theplurality of pixels PX are provided, and a first pad area PA1 providedaround the sensing area SA and the logic circuit LC. A plurality ofupper pads PAD are included in the first pad area PAL The plurality ofupper pads PAD may be connected to a memory chip MC, provided on thesecond layer 52, through a via or a wire. The second layer 52 mayinclude a memory chip MC and a dummy chip DC, and an encapsulation layerEN encapsulating the memory chip MC and the dummy chip DC. The dummychip DC may be omitted.

FIG. 4 illustrates a pixel array of an image sensor according to anexemplary embodiment of the inventive concept.

Referring to FIG. 4, a pixel array PA of an image sensor according to anexemplary embodiment includes a plurality of pixels PX. The plurality ofpixels may be connected to a plurality of row lines ROW1 to ROWm (ROW)and a plurality of column lines COL1 to COLn (COL). For example, a givenpixel of the pixels PX may be connected to a given row line of the rowlines ROW1 to ROWm and to a given column line of the column lines COL1to COLn. The image sensor may drive the plurality of pixels PX in unitsof the plurality of row lines ROW. As an example, time required fordriving a selected driving line among the plurality of row lines ROW andreading the reset voltage and the pixel voltage from pixels PX connectedto the selected driving line may be defined as one horizontal cycle. Theimage sensor may operate in a rolling shutter manner, in which aplurality of the pixels PX are sequentially exposed to light, or aglobal shutter manner in which a plurality of the pixels PX aresimultaneously exposed to light.

A reset voltage and a pixel voltage, output from each of the pluralityof pixels PX, may be converted into digital data and may be processed asraw data through predetermined signal processing. An image signalprocessor, mounted in the image sensor or an additional processorcommunicating with the image sensor, may generate a result imagedisplayed on a display or stored in a memory. Accordingly, differentresult images may generated from the raw data depending on performanceor a tuning method of the image signal processor. Thus, a user may beprovided with an optimal result image by improving performance of theimage signal processor or by precisely tuning the image signalprocessor.

When tuning of the image signal processor employs a method depending ona person's evaluation, it may be difficult to objectively and preciselytune the image signal processor. In an exemplary embodiment, theperformance of the image signal processor may be improved by providing amethod of modeling the image signal processor so there is no room forintervention of a person's subjective judgment. In addition, a user maybe provided with an optimal result image by tuning the image signalprocessor in consideration of the user's desire.

FIG. 5 is a block diagram of an electronic device according to anexemplary embodiment of the inventive concept.

Referring to FIG. 5, an electronic device 100 according to an exemplaryembodiment of the inventive concept includes an image sensor 110, aprocessor 120, a memory 130, and a display 140. The processor 120 maycontrol overall operation of the electronic device 100, and may beimplemented by a central processing unit (CPU), an application processor(AP), or a system on chip (SoC). In an exemplary embodiment, the imagesensor 110 and the image signal processor are mounted on a singleintegrated circuit chip.

The image sensor 110 may generate raw data in response to external lightand may transmit the raw data to the processor 120. The processor 120may include an image signal processor 121 configured to signal-processthe raw data to generate a result image. The image signal processor 121may adjust a plurality of parameters associated with the raw data andsignal-process the raw data according to the adjusted parameters togenerate a result image. The parameters may include two or more ofcolor, blurring, sharpness, noise, a contrast ratio, resolution, and asize. In an alternate embodiment, the parameters may include only one ofcolor, blurring, sharpness, noise, a contrast ratio, resolution, and asize. The result image, output by the image signal processor 121, may bestored in the memory 130 or may be displayed on the display 140.

The processor 120 may include a parameter optimization module 122. In anexemplary embodiment, the parameter optimization module 122 and theimage signal processor are mounted in a single integrated circuit. As anexample, the parameter optimization module 122 may adjust weights givento the plurality of parameters, and characteristics of the result image,output by the image signal processor 121, can be changed depending onthe adjusted weights. As an example, the parameter optimization module122 adjusts a color, one of the plurality of parameters, to output awarm-tone result image or a cold-tone result image from the same rawdata. For example, when a first weight is applied to the color parameterto generate a first weighted parameter, the image signal processor 121processing the raw data using the first weighted parameter outputs awarm-tone result image. For example, when a second weight different fromthe first weight is applied to the color parameter to generate a secondweighted parameter, the image signal processor 121 processing the rawdata using the second weighted parameter outputs a cold-tone resultimage.

In an exemplary embodiment, the weight, applied to the plurality ofparameters by the parameter optimization module 122, is determined by amodeling method performed in advance. The weight, applied to theplurality of parameters by the parameter optimization module 122, may beadaptively adjusted based on user feedback. As an example, a weight maybe determined by a modeling method using a machine learning model tosignificantly reduce a possibility of intervention of a person'ssubjective evaluation and to improve performance of the image signalprocessor 121 while accurately and objectively tuning the image signalprocessor 121.

FIG. 6 is a flowchart illustrating a method of modeling an image signalprocessor according to an exemplary embodiment of the inventive concept,and FIG. 7 illustrates a system providing a method of modeling an imagesignal processor according to an exemplary embodiment of the inventiveconcept.

Referring to FIG. 6, a method of modeling an image signal processoraccording to an exemplary embodiment includes capturing an image of asample subject to obtain sample data (S100). Referring to FIG. 7, asystem 200 for modeling an image signal processor may include anelectronic device 210, including an image sensor, a sample subject 220,and a computer device 230 in which a modeling method is executed.Although the electronic device 210 is illustrated as being a camera, itmay be replaced with another device including an image sensor. Inaddition, although the computer device 230 is illustrated as being adesktop computer, it may be replaced with another device executing themodeling method. According to an exemplary embodiment, the electronicdevice 210 and the computer device 230 are implemented as a singledevice.

The sample subject 220 may be a test chart. The sample subject 220 mayinclude a plurality of capturing regions 221 to 223 (regions ofinterest), which may be different from each other. As an example, afirst capturing region 221 may be a region in which people aredisplayed, a second capturing region 222 may be a region in which ablack-and-white pattern is displayed, and a third capturing region 223may be a region in which a color pattern is displayed. The sample data,obtained by the electronic device 210 capturing the sample subject 220,may be raw data. The raw data may be transferred to the computer device230 including an image signal processor (ISP) simulator. In anembodiment, the ISP simulator is capable of simulating different typesof image signal processors. For example, the ISP simulator could emulateone or more of the image signal processors processing the raw data togenerate an image. Emulating a given image signal processor processingthe raw data may include the given image signal processor processing theraw data using one or more parameters. For example, a given parametermay be settable to only certain values, where each setting has adifferent affect. For example, if the given parameter is settable toonly a first value or a second other value, emulating an image signalprocessor processing the raw data using the given parameter set to thefirst value could result in a first image, while an emulating the imagesignal processor processing the same raw data using the given parameterset to the second value could result in a second image different fromthe first image.

The computer device 230 sets parameters used in an operation of an imagesignal processor to respective sample values (S110). The computer device230 signal-processes the raw data using the image signal processorsimulator to emulate the image signal processor processing the sampledata using the sample values to generate a plurality of sample images(S120).

In the modeling method executed in the computer device 230, a pluralityof sample scores of evaluation items are obtained for each of theplurality of sample images (S130). The plurality of sample scores may bescores calculated from a plurality of evaluation items selected toevaluate each of the plurality of sample images. As an example, theplurality of evaluation items may include at least one of an imagecolor, resolution, a dynamic range, shading, sharpness, texture loss,and noise. For example, if only resolution is considered, the firstsample image has a low resolution and the second sample image has a highresolution, the first sample image could receive a lower score than thesecond sample image.

The computer device 230 stores the sample values for the parameters, thesample images, and sample scores, in a database (DB) (S140). Thedatabase may include a mapping of each parameter to a respective samplevalue. For example, the sample values, sample images, and sample scoresstored in the database (DB), may be used to train a machine learningmodel to infer the performance of the image signal processor or topredict the quality of an image that will be produced by the imagesignal processor when parameters having certain values are used duringprocessing of raw data.

FIGS. 8 to 10 illustrate a method of modeling an image signal processoraccording to an exemplary embodiment of the inventive concept.

FIG. 8 may be a schematic diagram of a system for modeling an imagesignal processor. Referring to FIG. 8, the system 300 includes asimulator 310, an evaluation framework 320, and a database 330.

The simulator 310 receives sample data 301, which is raw data obtainedby capturing an image of a sample subject such as a test chart. Thesimulator 310 may include a parameter generator 311, configured todetermine a plurality of sample values for a plurality of parameters 232used in an operation of an image signal processor, and an ISP simulator312 configured to simulate (or emulate) the image signal processoroperating on the sample data 392 using the sample values of theparameters 232. For example, the parameter generator 311 may determinesample values of the parameters 232 such as image color, blurring,noise, a contrast ratio, resolution, and size. At least one of theparameters may be classified into a plurality of detailed parametersaccording to an embodiment. For example, there may be a plurality ofdetailed parameters for noise and a plurality of detailed parameters forcolor.

The ISP simulator 312 may signal-process the sample data 301 using theplurality of sample values 332, determined for the plurality ofparameters by the parameter generator 311, to generate sample images331. Hereinafter, the operation of the simulator 310 will be describedin more detail with reference to FIG. 9.

Referring to FIG. 9, the parameter generator 311 generates a pluralityof sample values for first through sixth parameters. As an example, theparameter generator 311 may generate first through sixth sample setshaving different sample values for the first through sixth parameters.The first to sixth parameters are parameters used in an operation of theimage signal processor. The number and types of the parameters may bevariously changed. Similarly, the number of sample sets, generated bysetting the sample values for the parameters by the parameter generator311, may also be variously changed.

When the sample sets are determined, the ISP simulator 312 generates thefirst through sixth sample images 410 to 460 (400) by setting parametersto each of the sample sets and simulating (or emulating) the operationof the image signal processor on the sample data 301 using each of thesample sets. For example, the ISP simulator 312 may emulate the imagesignal processor processing raw data of the sample data 301 using the 6sample parameters set to their respective values in the first sample setto generate the first sample image 410, emulate the image signalprocessor processing the raw data using the 6 sample parameters set totheir respective values in the second sample set to generate the secondsample image 420, etc. In an exemplary embodiment, the sample images 400are images generated from the sample data 301 obtained by capturing animage of the same sample subject. Since the sample images 400 are imagesgenerated by the ISP simulator 312 by different sample sets, they mayhave different quality and/or characteristics.

Returning to FIG. 8, the evaluation framework 320 receives sampleimages, generated by the simulator 310, to evaluate the quality of thesample images. As an example, the evaluation framework 320 may evaluateeach of the sample images for a plurality of evaluation items and mayexpress a result of the evaluation as sample scores 333. In the exampleembodiment illustrated in FIG. 8, the evaluation framework 320 mayobtain sample scores 333 of evaluation items such as resolution 321,texture loss 322, sharpness 323, noise 324, a dynamic range 325, shading326, and a color 327, for each of the sample images. Hereinafter, thiswill be described in more detail with reference to FIG. 10.

Referring to FIG. 10, the evaluation framework 320 receives the sampleimages 400 to obtain sample scores 333 for a plurality of evaluationitems. As an example, sample scores for the first through thirdevaluation items may be obtained by evaluating each of the sample images400. For example, the first evaluation item could be resolution 321, thesecond evaluation item could be sharpness 323, and the third evaluationitem could be noise 324. The evaluation framework 320 may classify andstore the sample scores depending on the sample images 400. A lowestpoint and a highest point of each of the sample scores may varydepending on the sample items. For example, as shown in FIG. 10, thefirst sample image 410 includes a first score of 70.37 for the firstevaluation item, a second score of 62.29 for the second evaluation item,and a third score of 1979.25 for the third evaluation item.

When the evaluation by the evaluation framework 320 has completed, thedatabase 330 may be established. The database 330 includes sample images331 and sample values 332 of the plurality of parameters, generated bythe simulator 310, and sample scores 333 obtained by evaluating thesample images 331 for the plurality of evaluation items 321 to 327 bythe evaluation framework.

The sample images 331, the sample values 332 of the plurality ofparameters, and the sample scores 333, stored in the database 330, maybe used to train the machine learning model. The machine learning model,trained by data stored in the database 330, may be a model forpredicting the quality of a result image output by the image signalprocessor. Hereinafter, this will be described in more detail withreference to FIGS. 11 and 12.

FIG. 11 illustrates a system providing a method of modeling an imagesignal processor according to an exemplary embodiment of the inventiveconcept, and FIG. 12 illustrates a machine learning model employed in amethod of modeling an image signal processor according to an exemplaryembodiment of the inventive concept.

Referring to FIG. 11, a system 500 according to an exemplary embodimentmay operate in cooperation with a database 600. The database 600 may bea database established by the modeling method described with referenceto FIGS. 8 to 10, and may include sample images, sample values for aplurality of parameters, and sample scores for a plurality of evaluationitems.

In an exemplary embodiment, the machine learning model trainer 510trains a machine learning model 700 to predict a quality of an imageproduced by a given image signal processor using parameters have certainvalues using sample values 501 of the parameters and sample scores 502stored in the database 600. As an example, the sample values 501 of theparameters may be at least one of first to sixth sample sets set in thesame manner as described in the example embodiment with reference toFIG. 9. As an example, when the first sample set is selected, a firstsample score set in the example embodiment, illustrated in FIG. 10, maybe selected as the sample scores 502.

The machine learning model trainer 510 may input sample values, includedin the first sample set, to the machine learning model 700. In anexemplary embodiment, the machine learning model trainer 510 trains themachine learning model 700 until the output of the machine learningmodel 700 matches evaluation scores of the first sample score set or adifference between evaluation scores of the first sample score setbecomes less than or equal to a reference difference.

Referring to FIG. 12, the machine learning model 700 may be implementedby an artificial neural network (ANN). The machine learning model 700includes an input layer 710, a hidden layer 720, an output layer 730. Asan example, a plurality of nodes, included in the input layer 710, thehidden layer 720, and the output layer 730, may be connected to eachother in a fully connected manner. The input layer 710 includes aplurality of input nodes x₁ to x_(i). In an exemplary embodiment, thenumber of the input nodes x₁ to x_(i) corresponds to the number ofparameters. The output layer 730 includes a plurality of output nodes y₁to y_(j). In an exemplary embodiment, the number of the output nodes y₁to y_(j) corresponds to the number of evaluation items.

The hidden layer 720 includes first to third hidden layers 721 to 723,and the number of the hidden layers 721 to 723 may be variously changed.As an example, the machine learning model 700 may be trained byadjusting weights of the hidden nodes included in the hidden layer 720.For example, the first to sixth sample sets are input to the input layer710 and the weights of the hidden nodes, included in the hidden layer720, may be adjusted until values, output to the output layer 730,correspond to the first to sixth sample score set. Accordingly, afterthe training has completed, quality of a result image, output by theimage signal processor, may be inferred using the machine learning model700 when the parameters have predetermined values.

FIG. 13 is a flowchart illustrating a method of modeling an image signalprocessor according to an exemplary embodiment of the inventive concept.

Referring to FIG. 13, a method of modeling an image signal processoraccording to an exemplary embodiment of the inventive concept includessetting initial values for a plurality of parameters applied to an imagesignal processor (S200). The plurality of parameters applied to theimage signal processor may include a color, blurring, noise, a contrastratio, a resolution, and a size an image as parameters used in anoperation of the image signal processor.

Next, the initial values for the parameters are input to the machinelearning model (S210). The machine learning model may be a model trainedto predict the quality of the resulting image output by the image signalprocessor. An output of the machine learning model may vary depending onthe values of the parameters applied to the image signal processor. Atraining process of the machine learning model may be understood basedon the example embodiment described above with reference to FIGS. 11 and12.

Evaluation scores for a plurality of evaluation items are obtained usingthe output of the machine learning model (S220). As described above, themachine learning model is a model trained by the image signal processorto predict the quality of a result image generated by signal-processingraw data, and the output of the machine learning model corresponds tothe evaluation scores of a plurality of evaluation items. In anexemplary embodiment, the plurality of evaluation items may include acolor, sharpness, noise, resolution, a dynamic range, shading, andtexture loss of an image.

In the modeling method according to an exemplary embodiment, weightsapplied to the parameters are adjusted based on the obtained evaluationscores for the plurality of evaluation items (S230). As an example, eachof the evaluation scores may be compared with predetermined referencescores and, when there is an evaluation score which does not reach areference score, the weight is applied to at least one of the parametersmay be increased or decreased such that the corresponding evaluationscore may be increased. Alternatively, the evaluation score, output bythe machine learning model, may be compared with a reference score whilechanging a weight by a predetermined number of times.

FIGS. 14 to 17 illustrate a method of modeling an image signal processoraccording to an exemplary embodiment of the inventive concept.

FIG. 14 is a schematic diagram of a system for providing a method ofmodeling an image signal processor according to an exemplary embodimentof the inventive concept. Referring to FIG. 14, a system 800 accordingto an exemplary embodiment includes a parameter adjusting module 810, amachine learning model 820, and a feedback module 830.

The parameter adjusting module 810 may adjust values input to a machinelearning model 820. The machine learning model 820 may receiveparameters used in an operation of an image signal processor, and mayoutput evaluation scores indicating quality and/or characteristics of aresulting image generated by the image signal processor operatingdepending on values of the parameters. Accordingly, the parameteradjusting module 810 may adjust the values of the parameters used in theoperation of the image signal processor. For example, the parameteradjusting module 810 may adjust weights applied to the parameters. Wheninitial values of parameters 801 are input, the parameter adjustingmodule 810 may apply predetermined weights to the initial values of theparameters to generate weighted values of the parameters, and input theweighted values to the machine learning model 820.

The machine learning model 820 may be a model trained to predict thequality of the resulting image generated by the image signal processor.An output of the machine learning model 820 may correspond to anevaluation score of the evaluation items indicating quality of a resultimage. In an exemplary embodiment, the feedback module 830 compares anoutput of the machine learning model 820 with a target score of theevaluation items and transmits a result of the comparison to theparameter adjusting module 810. In an exemplary embodiment, theparameter adjusting module 810 adjusts weights applied to theparameters, with reference to a comparison result transmitted by thefeedback module 830. The parameter adjusting module 810 may adjust theweights applied to the parameters, a predetermined number of times oruntil the difference between the evaluation scores and the target scoresoutput by the machine learning model 820 is reduced to be less than orequal to the reference difference. When adjusting the weights hasfinished, optimized ISP parameters 802 (e.g., parameters set to optimalvalues) may be output from the system 800. The parameters set to theoptimal values may be used to tune an image signal processor. Theparameters set to the optimal values may be output to the image signalprocessor for storage on the image signal processor and then the imagesignal processor can use the parameters set to these values whenperforming a subsequent operation (e.g., process raw data to generate animage).

The system 800 may adjust the weights applied to the parameters, byconsidering feedback from a user of an electronic device in which animage signal processor is mounted. In this case, the system 800 may bemounted in the electronic device together with the image signalprocessor and may adaptively adjust the weights with reference to thefeedback from the user.

FIGS. 15 to 17 are provided to illustrate a method of modeling an imagesignal processor according to an exemplary embodiment of the inventiveconcept. Referring to FIG. 15, initial values may be set for first tosixth parameters. The initial values may be any values generated atrandom.

Referring to FIG. 16, a predetermined weight may be reflected on aninput layer IL to be input to the machine learning model. For example,the input layer IL may receive a plurality of input values, and theplurality of input values may correspond to parameters used in anoperation of the image signal processor. A weight may be applied to theplurality of input values, and the plurality of input values and theweight may be connected in a fully connected manner or a partiallyconnected manner. When the plurality of input values and the weightcorrespond to each other in the partially connected manner, the weightis not connected to at least one of the plurality of input values.

The machine learning model may output at least one output value to anoutput layer OL using a plurality of weight-given input values. Theoutput value may correspond to an evaluation score of an evaluation itemwhich may indicate the quality of the image generated by the imagesignal processor. The number of input values included in the input layerIL, and the number of output values included in the output layer OL, maybe variously changed according to exemplary embodiments.

FIG. 17 is a graph illustrating variation of evaluation scores y₁ to y₄depending on the number of times of training of the machine learningmodel. In the exemplary embodiment illustrated in FIG. 17, it is assumedthat the output layer OL outputs the evaluation scores y1 to y4 for fourevaluation items. However, the assumption is merely an exemplaryembodiment as a shape of the layer is not limited thereto.

When the machine learning model outputs first to fourth evaluationscores y1 to y4, the first to fourth evaluation scores y₁ to y₄ arecompared with the first to fourth target scores, respectively. At leastone of the weights, applied to the plurality of input values, may bechanged depending on a result of the comparison. In the exemplaryembodiment illustrated in FIG. 17, weights applied to hidden nodes of ahidden layer included in the machine learning model, are not adjustedwhile weights applied to the plurality of input values in the inputlayer IL of the machine learning model, are adjusted.

As training is repeated while changing at least one of the weights, thefirst to fourth evaluation scores y₁ to y₄ output by the machinelearning model, may be approximated to each of the first to fourthtarget scores. At least one of the weights may be adjusted until apredetermined number of times of training completes or until adifference between the first to fourth evaluation scores y₁ to y₄ andthe first to fourth target scores is reduced to be less than a referencedifference. When the number of times of training completes or thedifference between the first to fourth evaluation scores y₁ to y₄ andthe first to fourth target scores is reduced to less than the referencedifference, weights may be determined. The determined weights may beassigned to input values of the input layer IL, corresponding toparameters used in an operation of the image signal processor, in thefully connected manner or the partially connected manner.

Raw data, obtained by capturing an image of a sample subject, may beinput to the image signal processor to tune the image signal processor,and the image signal processor may be tuned to satisfy predeterminedevaluation conditions output by the image signal processor. In thiscase, since the image signal processor is tuned using the raw dataobtained by capturing of an image of the sample subject, a relativelylong time may be required. In addition, when the tuning depends on aperson's objective evaluation, it may be difficult to objectively andprecisely tune the image signal processor.

Meanwhile, in at least one exemplary embodiment of the inventiveconcept, image data obtained by capturing an image of at least onesample subject, is processed by the image signal processor simulatoraccording to sample values of various parameters to generate sampleimages. Sample scores, obtained by evaluating the sample images, andsample values of the parameters may be stored in a database. Since thesample scores and sample values of the parameters stored in the databaseare numerical items, an effect of a person's subjective evaluation maybe significantly reduced. In addition, a machine learning model trainedto receive the sample values of the parameters and to output samplescores, may be prepared. Weights, applied to the parameters, may beadjusted such that evaluation scores output by the machine learningmodel receiving initial values of the parameters, reach target scores.

In at least one exemplary embodiment, an image sensor processor is tunedby adjusting weights applied to parameters used in an operation of theimage signal processor, with numerical items, and an effect of aperson's subjective evaluation may be significantly reduced toobjectively and precisely tune the image signal processor. Additionally,the image signal processor may be adaptively tuned depending on a userby considering an end-user's desire in processes of comparing theevaluation scores outputted by the machine learning model, with targetscores and adjusting weights of the parameters.

FIG. 18 is a block diagram of an electronic device according to anexemplary embodiment of the inventive concept.

An electronic device 900 according to an exemplary embodimentillustrated in FIG. 18 includes a display 910, an image sensor 920, amemory 930, a processor 940, and a port 950. The electronic device 900may further include a wired/wireless communications device and a powersupply. Among components illustrated in FIG. 18, the port 950 may beprovided for the electronic device 900 to communicate with a video card,a sound card, a memory card, and a universal serial bus (USB) device.The electronic device 900 may conceptually include all devices, whichemploy the image sensor 920, in addition to a smartphone, a tabletpersonal computer (PC), and a digital camera.

The processor 940 may perform a specific operation, command, or task.The processor 940 may be a central processing unit (CPU) or a system onchip (SoC), and may communicate with the display 910, the image sensor920, and the memory 930 as well as other devices connected to the port950 through a bus 960.

The processor 940 may include an image signal processor 941. The imagesignal processor 941 generates a result image using raw data generatedby the image sensor 920 capturing an image of a subject. The processor940 may display the result image generated by image signal processor 941on the display 910 and may store the result image in memory 930.

The memory 930 may be a storage medium configured to store datanecessary for an operation of the electronic device 900 or multimediadata. The memory 930 may include a volatile memory such as random accessmemory (RAM) or a nonvolatile memory such as a flash memory. The memory930 may also include at least one of a solid state drive (SSD), a harddisk drive (HDD), and an optical drive (ODD) as a storage device.

The memory 930 may include a machine learning model 931 such as themachine learning model 700. The machine learning model 931 may receiveparameters used in an operation of the image signal processor 941, andmay output evaluation scores of evaluation items indicating a quality ofthe result image generated by the image signal processor 941 using theparameters. As an example, the parameters input to the machine learningmodel 931 may include a color, blurring, noise, a contrast ratio, aresolution, and a size of an image. The evaluation scores, output by themachine learning model 931, may correspond to evaluation items such as acolor, sharpness, noise, a resolution, a dynamic range, shading, andtexture loss of the image.

The electronic device 900 may adaptively adjust weights applied to theparameters used in the operation of the image signal processor 941,using the machine learning model 931. In an exemplary embodiment, theelectronic device 900 does not train the machine learning model 931itself and merely adjusts the weights applied to the parameters in afront end of an input layer of the machine learning model 931. Thus, theimage signal processor 941 may be tuned for a user without great burdenof an arithmetic operation.

FIGS. 19A and 19B illustrate examples of electronic devices that mayinclude the electronic device 900.

Referring to FIGS. 19A and 19B, an electronic device 1000 according toan exemplary embodiment is a mobile device such as a smartphone.However, the electronic device 1000 is not limited to a mobile devicesuch as a smartphone. For example, and the electronic device 100 may beany device including a camera which captures an image.

The electronic device 1000 includes a housing 1001, a display 1002, andcameras 1005 and 1006. In an exemplary embodiment, the display 1002substantially covers an entire front surface of the housing 1001 andincludes a first region 1003 and a second region 1002, depending on anoperating mode of the electronic device 1000 or an application which isbeing executed. The display 1002 may be provided integrally with a touchsensor configured to sense a user's touch input.

The cameras 1005 and 1006 may include a general camera 1005 and atime-of-flight (ToF) camera 1006. The general camera 1005 may include afirst camera 1005A and a second camera 1005B. The first camera 1005A andthe second camera 1005B may be implemented with image sensors havingdifferent angles of view, different aperture values, or a differentnumber of pixels. Due to a thickness of the housing 1001, it may bedifficult to employ a zoom lens for adjusting an angle of view and anaperture value in the general camera 1005.

Accordingly, the first camera 1005A and the second camera 1005B, havingdifferent angles of view and/or different aperture values, may providean image capturing function satisfying user's various needs.

The ToF camera 1006 may be combined with an additional light source togenerate a depth map. The ToF camera 1006 may provide a face recognitionfunction. As an example, the ToF may operate in combination with aninfrared light source.

Referring to FIG. 19B illustrating a back surface of the electronicdevice 1000, a camera 1007 and a light emitting unit 1008 may bedisposed on the rear surface. Similar to the camera 1005 disposed on afront surface of the electronic device 1000, the camera 1007 includes aplurality of cameras 1007A to 1007C having at least one of differentaperture values, different angles of view, and a different number ofpixels of the image sensor. The light emitting unit 1008 may employ alight emitting diode (LED) as a light source and may operate as a flashwhen capturing images using the camera 1007.

As described with reference to FIGS. 19A and 19B, an electronic device1000, having two or more cameras 1005 to 1007 mounted therein, mayprovide various image capturing functions. An image signal processor,mounted in the electronic device 1000, needs to be appropriately tunedto improve the quality of a result image captured by the cameras 1005 to1007.

The image signal processor, mounted in the electronic device 1000, mayprocess raw data generated by the cameras 1005 to 1007 depending onvalues of a plurality of parameters to generate a result image. Qualityor characteristics of the result image may depend on the values of theparameters, applied to the image signal processor, in addition to theraw data. In an exemplary embodiment, weights are applied to theparameters used in an operation of the image signal processor togenerated weighted parameters, and the quality and characteristics ofthe result image are improved by adjusting the weights.

Alternatively, weights are applied to the parameters used in anoperation of the image signal processor to generate weighted parameters,and a user of the electronic device 1000 adjusts the weights to generatea preferred result image. For example, the electronic device 1000 maydirectly receive feedback from the user to adjust the weights applied tothe parameters. Alternatively, a color, sharpness, and a contrast ratioof the user's preferred image may be accumulated depending on acapturing site (e.g., the location where an image of the subject wascaptured), a capturing time (e.g., a time when the image of the subjectwas captured), and type of a captured subject, and thus, weights of theparameters, applied to the image signal processor, may be changed.

As an example, when the user prefers low sharpness and warm colors forimages on which people are captured outdoors on a sunny day, theelectronic device 1000 may adjust the weights applied to the parametersin a front end of an input layer of a machine learning model, such thatamong the evaluation scores output by an embedded machine learningmodel, sharpness and a color are adjusted toward a user's preference.The adjusted weights may be stored in a memory, and may be applied tothe parameters of the image signal processor when a capturingenvironment, in which a person is selected as a subject outdoors on asunny day, is recognized.

FIGS. 20 and 21 illustrate an operation of an electronic deviceaccording to an exemplary embodiment of the inventive concept.

FIG. 20 is a raw image corresponding to an image before an image signalprocessor signal-processes raw data, and FIG. 21 is a result imagegenerated by signal-processing raw data by an image signal processor. Inthe exemplary embodiments illustrated in FIGS. 20 and 21, the raw imagemay exhibit poorer noise characteristics than the result image. Forexample, certain weights applied to parameters of an image signalprocessor, may be set to values which improve noise characteristics. Forexample, other weights applied to parameters of an image signalprocessor, may be determined to be values deteriorating noisecharacteristics depending on user setting or a capturing environment.

As described above, according to an exemplary embodiment, a plurality ofparameters that determine operating characteristics of an image signalprocessor, may be tuned using a machine learning model. Weights for theplurality of parameters, applied to the image signal processor, may bedetermined using the machine learning model such that the image signalprocessor achieves optimal performance. Accordingly, the image signalprocessor may be objectively and precisely tuned, as compared with aconventional manner in which a person manually tunes the image signalprocessor. In addition, weights applied to parameters may be adjusted byconsidering feedback received from a user of an electronic device inwhich an image signal processor is mounted. Thus, an image signalprocessor optimized for the user may be implemented.

While exemplary embodiments of the inventive concept have been shown anddescribed above, it will be apparent to those skilled in the art thatmodifications and variations can be made without departing from thescope of the present inventive concept.

1. A method of training a machine learning model to predict optimalvalues for a plurality of parameters used in an operation of an imagesignal processor, comprising: capturing an image of a sample subject toobtain sample data; generating a plurality of sets of sample values forthe plurality of parameters; emulating the image signal processor (ISP)processing the sample data according to each of the sets to generate aplurality of sample images; evaluating each of the plurality of sampleimages for a plurality of evaluation items to generate respective samplescores; and training the machine learning model to predict the optimalvalues using the sample values and the sample scores.
 2. The method ofclaim 1, wherein the plurality of parameters include at least two of acolor, blurring, noise, a contrast ratio, a resolution, and a size of animage.
 3. The method of claim 1, wherein the plurality of evaluationitems include at least two of a color, sharpness, noise, a resolution, adynamic range, shading, and texture loss of an image.
 4. The method ofclaim 1, wherein the plurality of sets of the sample values include afirst sample set and a second sample set for the plurality ofparameters, and the plurality of sample images include a first sampleimage, corresponding to the first sample set, and a second sample imagecorresponding to the second sample set.
 5. The method of claim 4,wherein the sample scores comprise a first sample score set obtainedfrom the first sample image, and a second sample score set obtained fromthe second sample image.
 6. The method of claim 1, the training of themachine learning model comprising: inputting initial values, for theplurality of parameters, to the machine learning model to obtainevaluation scores for the plurality of evaluation items; and adjustingweights, applied to the plurality of parameters, such that theevaluation scores satisfy predetermined reference conditions.
 7. Themethod of claim 6, further comprising: inputting raw data to the imagesignal processor, having the plurality of parameters to which theweights are applied, to generate a result image when an image sensorcaptures a subject to generate the raw data.
 8. The method of claim 6,wherein the machine learning model is implemented as an artificialneural network.
 9. The method of claim 6, wherein the weights and theplurality of parameters are connected in a partially connected manner.10. The method of claim 1, wherein the sample subject includes aplurality of different subjects.
 11. A method of predicting optimalvalues for a plurality of parameters used in an operation of an imagesignal processor, comprising: inputting initial values for the pluralityof parameters to a machine learning model including an input layerhaving a plurality of input nodes, corresponding to the plurality ofparameters, and an output layer having a plurality of output nodes,corresponding to a plurality of evaluation items extracted from a resultimage generated by the image signal processor; obtaining evaluationscores for the plurality of evaluation items using an output of themachine learning model; adjusting weights, applied to the plurality ofparameters, based on the evaluation scores; and determining the optimalvalues using the adjusted weights.
 12. The method of claim 11, whereinat least some of the weights are adjusted by a user of a device in whichthe image signal processor is mounted.
 13. The method of claim 11,wherein the evaluation scores are obtained while adjusting the weights,and the adjustment of the weights completes when each of the evaluationscores satisfies predetermined reference conditions.
 14. The method ofclaim 11, wherein the evaluation scores are obtained while adjusting theweights a predetermined number of times.
 15. The method of claim 11,further comprising: tuning the image signal processor using the optimalvalues.
 16. The method of claim 11, wherein at least some of the weightshave different values depending on a subject captured by the imagesensor.
 17. The method of claim 11, further comprising: generating aplurality of sets of sample values for the plurality of parameters;emulating the image signal processor (ISP) processing the sample dataaccording to each of the sets to generate a plurality of sample images;evaluating each of the plurality of sample images for a plurality ofevaluation items to generate respective sample scores; and training themachine learning model using the sample values and the sample scores.18. An electronic device comprising: an image signal processorconfigured to process raw data, output by an image sensor, depending ona plurality of parameters to generate a result image; and a parameteroptimization module including a machine learning model, receiving samplevalues for the plurality of parameters and outputting a plurality ofsample scores indicating quality of sample images, the sample imagesbeing generated by the image signal processor processing the raw databased on the sample values, the parameter optimization module beingconfigured to determine weights, respectively applied to the pluralityof parameters, using the machine learning model, wherein the imagesignal processor applies the weights to the plurality of parameters togenerate a plurality of weighted parameters and generates the resultimage by processing the raw data using the weighted parameters.
 19. Theelectronic device of claim 18, wherein the image signal processor andthe parameter optimization module are mounted on a single integratedcircuit chip.
 20. The electronic device of claim 18, wherein the imagesignal processor and the image sensor are mounted on a single integratedcircuit chip. 21-22. (canceled)